James Bryant
   |
Posted - 2011.01.05 20:41:00 -
[1]
Edited by: James Bryant on 05/01/2011 20:42:48
Kallahar,
I applaud your efforts here. As a former dev of Eve Market Reports (if anyone remembers it, you'll remember it didn't last long), I'll throw out there some of the lessons I learned.
1.
Very, very quickly, you're going to end up with a truly impressive amount of data. Many Gb, in fact, and most likely more or less in a single or very few tables. Figuring out the most efficient queries to select this data is paramount. Any small scaling problems will very quickly become huge scaling problems. Utilize db indices to their fullest extent and build your schema very carefully.
2.
When you start getting numbers for your historical data, you will be needing to do a lot of math and queries per page. You really need a messaging and task system to automate creating tables of statistical and historical calculations (avg,sum,median,etc) on a daily or even hourly basis. There is no way to sustain doing all these calcs on a per page load basis (it will work in the beginning, but not next month). Cron is not ideal here, but if that's all you have, it can sorta work for a while, although I think that's what evemetrics 1.0 was doing and you all remember how slow that site could be.
3.
Build on a framework. A framework gives you a caching system, which will be a necessity. It also can handle user registration, logins, models, views, controllers, etc. By no means should you not use a caching system of some kind, at the very least. I've been using Yii, and I like it a lot, although it is very young and the documentation can be a bit spotty. Zend is good also.
4.
You won't be able to last on shared hosting for very long, if that's what you are on now. This type of site does a lot of heavy lifting, both in the foreground and in the background. Utilize AJAX as much as possible for slow data fetches that will unnecessarily delay page loads. Prepare for the time where a dedicated server will become necessary (it will become necessary).
I know this sounds like a lot of ballyhooing, but these are things to be aware of and are challenges that will arise faster than you expect. I've been there already. There is no shame in scaling functionality back from evemetrics. 1.0 was written without an awareness of these challenges, and needed to be rewritten from the ground up for 2.0, and was done so properly having encountered those problems with those strategies mentioned above.
Eve Market Reports failed ultimately because of a lack of time and a lack of preparation for the high software and hardware requirements of the project.
|